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Introduction 

Objective  measurement  of  pilot  proficiency  has  long  been  a problem  for 
researchers  attesting  to  improve  pilot  screening  and  training  methods.  Auto- 
mated performance  measurement  has  been  considered  the  only  .ompletely  objective 
method.  The  approaches  have  run  the  gamut  from  film  or  video  recording  (Wood 
and  Hagln,  1974)  to  Instrumented  aircraft  (Knoop  and  Welde,  1973)  and  advanced 
simulators  (Waag,  1975).  Unfortunately,  few  of  these  efforts  have  resulted  in 
readily  available  measurement  systems  or  schema:  fllm/video  techniques  are 
difficult  to  evaluate  and  score  and  instrumented  aircraft  and  advanced  simulators 
are  expensive. 

As  a result,  flying  training  research  still  relies  heavily  on  subjective 
Instructor  and  check-pilot  grades.  Although  several  training  research  studies 
carried  out  In  recent  years  have  used  these  kinds  of  subjective  measures 
successfully  (e.g.,  Reid,  1974),  many  problems  arise  for  which  more  precise, 
objective  data  are  required  if  meaningful  discriminations  are  to  be  made. 

The  alternative  to  automated  measurement — controlled  human  observation 
and  recording  using  standardized  and  structured  methods  and  material s--has 
provided  useful  data  when  the  recording  instrument  has  been  properly  designed 
and  the  observer-recorder  carefully  trained  in  its  use  (Ericksen,  1952).  While 
several  investigators  have  used  trained  observers  successfully  (e.g.,  Prophet 
and  Jolley,  1969;  Koonce,  1974),  many  others  have  avoided  the  technique 
because  of  the  effort  and  time  required  to  develop  an  effective  recording  form 
and  to  train  observers  to  an  acceptable  level  of  recording  reliability.  Indeed, 
frequently  the  aircraft  and  instructor  time  needed  were  not  available. 

Clearly,  observer  recording  as  a pilot  proficiency  measurement  tool  would 
become  more  attractive  if  techniques  were  found  to  reduce  the  development  time 
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and  effort  usually  required.  This  report  documents  the  development  of  an 
observer-recording  form  for  the  Instrument  training  maneuver  Vertical  S-A  In 
which  the  unique  record/playback  features  of  the  Advanced  Simulator  for  Pilot 
Training  (ASPT)  were  used  to  increase  the  efficiency  of  format  construction, 
reliability  assessment,  and  observer  training.  Implications  of  this  work  for 
a revitalized  Interest  in  observer  recording  techniques  as  low  cost  objective 
measures  of  pilot  skill  are  discussed. 


Method 


Maneuver  Selection  and  Analysis 


The  Vertical  S-A  was  selected  as  representative  of  the  many  instrument 


training  exercises  used  to  teach  some  of  the  skills  required  for  actual 
weather  flying.  The  student  first  establishes  straight  and  level  flight  at  a 
prescribed  altitude  and  airspeed.  Then  he  makes  a transition  to  a constant 
rate  climb  of  1000  ft/min.  After  climbing  exactly  1000  ft,  he  makes  a transi 
tion  to  a 1000  ft/min,  descent,  descends  1000  ft,  and  concludes  the  maneuver 


by  leveling  off  at  his  starting  altitude. 

Brecke,  Gerlach,  and  Schmid  (1976)  showed  that  the  maneuver  may  be 
envisioned  as  a series  of  steady  states  (straight  and  level,  climb,  etc.), 
separated  by  a series  of  transitions  from  one  steady  state  to  the  next.  Their 
maneuver  segmentation  provided  a convenient  set  of  discrete  piloting  behavior 
elements  during  which  an  observer  could  be  expected  to  observe  and  record 
deviations  from  the  prescribed  parameters.  Figure  1 graphically  illustrates 
the  flight  path  of  the  Vertical  S-A  and  the  maneuver  segments  defined. 


I 
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Figure  1.  The  Vertical  S-A  maneuver 
Measurement  Strategy 

A number  of  decisions  had  to  be  made  at  the  outset  concerning  the  types 
of  measures  to  be  used  and  the  role  of  the  observer- recorder  during  in-flight 
data  gathering.  The  first  decision  was  to  collect  data  and  establish  perfor- 
mance scores  by  maneuver  segments  rather  than  by  time  sampling.  This  approach 
is  consistent  with  a logical  analysis  of  the  maneuver.  It  eliminates  some  of 
the  difficulties  found  in  time  sampling,  e.g. , at  a given  point  in  time  differ- 
ent pilots  could  be  at  different  points  in  the  maneuver  and  the  resulting 
values  would  not  be  comparable. 

The  second  decision  was  to  obtain  maximum  deviation  scores  (where  appro- 
priate), accumulated  over  the  entire  segment.  This  was  done  because  pilot 
observer-recorders  are  more  accustomed  to  observing  deviations  from  prescribed 
parameters  than  they  are  with  observing  those  obtained  by  time  sampling. 
Although  detail  in  measurement  is  lost  through  this  approach,  experience  has 


also  shown  that  time  sampling  is  difficult  for  pilots  (Ericksen,  1947). 

The  third  decision  was  to  limit  instructor  pilots  (IPs)  to  recording 
observed  values  only  and  not  to  make  any  segment  or  overall  maneuver  performance 
appraisals.  This  was  done  to  avoid  the  possibility  that  the  IPs'  recording 
accuracy  might  be  influenced  by  attention  to  other  performance  variable  they 
might  customarily  use. 

Recording  Format  Development 

Performance  measurement  staff  scientists  of  the  Air  Force  Human  Resources 
Laboratory/Flying  Training  and  expert  instructor  pilots  from  Williams  AFB, 
Arizona  assisted  us  in  tne  development  of  a first  draft  of  a maneuver  recording 
booklet.  It  contained  both  instructions  for  time  intervals  or  points  at  which 
to  make  observations  and  scales  on  which  to  record  the  deviations  from  normal 
as  they  occurred.  In  general,  it  followed  the  style  of  the  manual  developed 
by  Koonce  (1974). 

The  pages  were  constructed  so  that  as  each  page  was  turned,  the  instruc- 
tions were  on  the  left  and  the  recording  scales  on  the  right.  Both  instructions 
and  their  corresponding  scales  were  serially  numbered.  The  booklet  fit  an 
instructor  pilot's  knee  clipboard.  A brief  tryout  in  the  ASPT  revealed 
several  format  and  observer-workload  problems  that  needed  correction. 

Following  revision,  the  booklet  (Aopendix  A)  was  given  a rigorous  evalua- 
tion in  the  ASPT.  Three  instructor  pilots  (IPs)  were  used:  one,  in  the  left 
seat  of  the  simulator  cockpit,  flew  the  Vertical  S-A  maneuvers;  the  second  sat 
in  the  right  seat  as  an  observer-recorder;  the  third  was  stationed  at  the  ASPT 
console  in  front  of  the  cockpit  repeater  instruments.  As  a series  of  maneuvers 
was  flown,  the  experimenter  noted  the  major  difficulties  encountered.  It 
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quickly  became  apparent  that  the  observers  were  having  difficulty  keeping  up 
with  the  maneuver  and  were  losing  data.  For  example,  having  to  write  down  an 
actual  value  was  found  to  be  more  difficult  than  marking  a scale.  In  addition, 
several  of  the  instructions  for  recording  were  so  confusing  that  they  caused 
errors. 

The  revisions  were  made  and  the  booklet  was  again  tested  in  the  ASPT.  As 
before,  three  IPs  were  used  (one  from  the  first  group  plus  two  new  ones).  One 
flew  the  maneuvers  while  the  other  two— one  In  the  cockpit  and  one  at  the 
operator  console--did  the  recordinq.  The  IP  at  the  controls  was  Instructed  to 
fly  a total  of  twelve  Vertical  S-As  representative  of  the  range  of  performances 
expected  from  students.  After  six  such  maneuvers,  the  two  observers  changed 
places  so  that  each  observed  the  same  number  of  maneuvers  in  the  cockpit  and 
at  the  console. 

Observer-observer  agreement,  although  improved,  was  still  inadequate.  The 
experimenter's  observations  were  confirmed  during  debriefing:  too  much 
detailed  recording  behavior  was  required,  so  that  the  observers  were  missing 
data  and  falling  behind.  The  following  specific  problems  and  concerns  were 
noted: 

(1)  The  attitude  indicator  was  a source  of  difficulty.  Not  only 
was  it  hard  to  read,  but  the  effort  expended  caused  delay  and 
error  in  reading  other  instruments.  Since  attitude  information 
is  reflected  in  the  vertical  velocity  indicator  readouts,  it 

was  decided  not  to  record  attitude  indicator  readings  henceforth. 

(2)  Power  settings  were  also  troublesome.  Since  they  were  included 
on  the  form  mainly  as  indices  of  segment  transitions  and  not 

as  performance  indicators  per  se , they  were  dropped  from  the  form 
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and  replaced  by  a simple  notation  indicating  whether  or  not  the 
subject  "led  with  power". 


(3)  The  straight-and-level  segment  was  also  deleted,  since  IPs  and 
the  experimenter  agreed  that  it  would  not  provide  much  useful 
information  for  the  planned  training  experiments. 

(4)  Maneuver  execution  time  had  been  included  as  a potentially 
valuable  diagnostic  Index  (Shipley,  1976).  However,  the  IPs 
and  research  staff  were  experiencing  difficulty  defining  and 
judging  when  the  maneuver  began  and  ended.  This  problem  was 
eliminated  by  re-defining  the  start  as  "time  at  which  the 
altimeter  passes  through  15,200  ft  at  start  of  climb,"  and 
completion  as  "tine  at  which  the  altimeter  passes  through 
15,200  on  descent."  This  provided  an  operational  definition 
of  maneuver  time  which  was  extremely  simple  and  which  resulted 
in  almost  no  loss  of  meaningful  information. 

(5)  Photo-reduction  to  fit  the  IP  kneeboard  was  discontinued  because 
the  pilots  expressed  a preference  for  a larger  format  and 
heavier  card  stock. 

The  resulting  booklet  is  shown  in  Appendix  B. 

Observer  Training 

The  record-playback  feature  of  the  ASPT  was  used  to  establish  base- 
line standard  Vertical  S-A  maneuver  performances.  An  IP  flew  a series  of 
S-A  maneuvers  simulating  the  range  of  performances  expected  of  students.  Twelve 
such  maneuvers  representing  poor,  average. and  good  student  performances  were 
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recorded  and  stored  on  both  computer  disc  and  magnetic  tapeJ 

Two  experimenters  used  the  recording  booklets  to  evaluate  the  twelve 
maneuvers  as  they  were  played  back.  By  playing  each  maneuver  several  times 
at  both  regular  and  half  speed,  and  observing  individual  parameters  carefully, 
a very  close  approximation  to  the  true  values  was  achieved.  These  values  were 
accepted  as  the  criteria  for  instructor/observer  training. 

The  two  instructor  pilots  who  were  to  serve  as  observers  in  the  planned 
experiment  were  trained  as  follows: 

(1)  The  IP  was  given  a copy  of  the  performance  measure  to  examine.  Any 
questions  he  had  concerning  the  form  or  use  of  the  measure  were 
answered  before  the  formal  training  session  began. 

(')  The  IP  was  first  seated  at  the  ASPT  console,  in  front  of  the 
Instrument  panel.  He  was  given  a stop-watch,  pen,  and  a set  of 
booklets  for  recording  performances. 

(3)  The  twelve  trials  were  randomized  for  order  of  presentation.  While 
the  first  maneuver  was  being  played  back,  the  IP  recorded  the 
values  in  the  booklet. 

(4)  When  the  playback  of  the  maneuver  was  finished,  the  values  recorded 
by  the  IP  were  checked  against  the  true  values  previously  ootalned 
and  verified. 

(5)  The  maneuver  was  replayed  for  the  IP  to  show  him  where  he  had 
encountered  difficulty  and  to  suggest  ways  to  improve  performances. 


It  had  been  hoped  that  an  objective  evaluation  of  these  performances 
could  be  obtained  using  the  ASPT  automated  performance  measurement  system 
to  provide  the  "true"  maximum  range  deviation  and  maximum  deviation  scores 
for  the  various  maneuver  segment  parameters  of  interest.  However,  the 
existing  ASPT  software  did  not  permit  both  evatuation  of  real-time  flight 
and  maneuver  recording,  nor  did  it  allow  evaluation  of  a st  'd  maneuver 
while  being  played  back. 


(6)  This  procedure  continued  until  the  IP  was  able  to  make  recordings 
which  correlated  highly  with  the  experimenter  verified  values  and 
was  able  to  make  "complete"  recordings  (l.e.,  without  missing  values 
for  any  of  the  parameters). 

(7)  The  IP  was  then  given  an  opportunity  to  make  a number  of  recordings 
from  the  co-pilot  scat  In  the  ASPT  cockpit  to  be  sure  that  he  per- 
formed as  well  there  as  at  the  console. 

After  the  two  IPs  had  been  initially  trained,  a few  minor  changes  were 
made  to  simplify  the  materials  and  procedures.  Since  the  IPs  were  now 
thoroughly  familiar  with  the  points  at  which  measures  were  to  be  recorded, 
the  textual  and  diagrammatic  portions  of  the  booklet  seemed  superfluous.  The 
maneuver  segment  recording  pages  were  photo-reduced  so  that  all  five  segments 
would  fit  on  one  8-1/2"  x 11"  page.  This  eliminated  page-turning,  which  had 
been  a minor  nuisance  during  training.  The  performance  measures  were  printed 
on  a heavy  card  stock  to  provide  a fi  rm  marking  surface  (Appendix  C).  The 
two  IPs  were  given  further  tryout  on  this  final  version  to  assure  that  the 
format  simplification  had  not  changed  recording  reliability. 

Administrati  on 

The  performance  recording  fo’-m  was  then  used  tc  gather  data  on  three  groups 
of  subjects,  each  of  which  had  received  a different  instructional  treatment  for 
flying  the  Vertical  S-A.  All  of  the  subjects  had  previously  been  trained  in 
the  T-4  instrument  trainer  on  the  fundamental  techniques  of  aircraft  pitch, 
bank,  and  power  control,  but  had  not  yet  received  specific  Instruction  In 
flying  a Vertical  S-A. 
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Each  subject  flew  six  trials  In  accordance  with  the  following  Instructions 
from  his  IP-observer: 

"I  will  start  you  out  at  15000  ft,  160  knots  on  a heading  of 
180°.  You  will  have  a few  minutes  to  warm  up  by  flying  straight  and 
level  and  then  I will  ask  you  to  perform  a Vertical  S-A.  You  then 
may  start  when  you  are  ready.  I will  record  how  well  you  do  on  this 
form  (showed  recording  Instrument).  When  you  finish  each  trial,  I 
will  give  you  time  to  stabilize  straight  and  level  on  heading, 
altitude,  and  airspeed  before  telling  you  to  start  another  trial. 

There  will  be  six  in  all. " 

Scoring 

Each  maneuver  segment  proviJed  either  a maxima  range  of  deviation  (for 
heading,  airspeed,  and  vertical  velocity),  or  a single  deviation  score  (for 
maximuti  altitude  and  maneuver  time).  Since  pilot  behavior  Is  conventionally 
described  In  terms  of  such  variables  as  heading,  altitude,  and  airspeed  control. 
It  was  decided  to  combine  the  segment  values  for  each  of  these  variables  by 
averaging  across  segments  to  obtain  trial  scores.  No  attempt  was  made,  however, 
to  combine  these  separate  values  Into  a single  value  for  the  complete  maneuver. 

Results 

Evidence  of  learning  during  the  six  trials  was  considered  to  be  a necessary 
and  sufficient  Indicator  of  measurement  sensitivity.  Tables  1 and  2 show  the 
mean  delation  ranges  for  heading  and  airspeed  for  the  three  treatment  groups. 
The  absolute  values  reflect  fairly  good  performance,  even  cn  the  early  trials. 
Analyses  of  variance  of  these  data  indicate  that  Improvement  over  trials  was 
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Mean  Heading  Deviation  Ranges  In  Degrees 
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Table  3 

Mean  Vertical  Velocity  Deviation  Ranges  In  Feet 
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Mean  T1n>e  Maximum  Deviation  Score  In  Seconds 
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was  significant:  £(5,  135)  * 3.34,  £<.01  for  heading  and  £(5,  135)  * 11  .53, 

£ < .01  for  airspeed. 

Tables  3 and  4 an  mean  deviation  ranges  for  vertical  velocity  control 
and  maneuver  times.  Although  Inspection  shows  that  Treatment  6roup  1 was  not 
improving,  analyses  of  variance  of  these  data  also  revealed  significant  trials 
effects  for  Groups  II  and  III:  £(5,  135)  • 3.60,  £<.05  for  time.  This  Indi- 
cates that  these  measures,  too,  were  reflecting  improvement  with  practice. 

There  was  a significant  treatment  effect  and  interaction  due  to  the  failure  of 
Group  1 to  Improve  over  trials:  £(10,  135)  = 5.44,  £<.01.  This  effect  is 
explained  In  Brecke  et  al.  (1976). 

Discussion  and  Conclusions 

That  trained  observers  can  use  a well-designed  recording  form  as  a means 
of  objectifying  pilot  performance  measurement  and  of  providing  data  useful  for 
training  methods  research  has  long  been  established.  One  more  demonstration 
Is  of  little  significance.  What  is  of  importance,  however,  is  the  methodologi- 
cal advance  In  observer  training  and  in  pre-determining  measurement  reliability 
through  the  use  of  the  re  cord- pi  ay  back  feature  of  modem,  advanced  digital 
flight  simulators.  This  makes  It  possible  to  train  observers  to  a desired 
level  of  recording  reliability  In  lieu  of  having  to  accept  post  hoc  estimates, 
as  has  been  the  case  with  all  earlier  exploration  of  observer-recorded 
measures.  When  one  knows  precisely  what  the  true  value  should  be  for  para- 
meters of  high  Interest  and  is  able  to  fly  a given  maneuver  set  over  and  over, 
It  Is  possible  to  train  each  observer  to  the  same  level  of  recording  accuracy  — 
a degree  of  control  over  recording  objectivity  and  reliability  not  previously 
possible.  As  a result,  researchers  are  no  longer  confined  to  after-the-fact 
correlational  reliability  estimations. 
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As  in  this  study,  the  observer  recording  approach  to  objective  pilot  per- 
formance measurement  has  been  most  successful  in  instrument  flight  training 
where  both  the  maneuver  and  criteria  can  be  conveniently  described  in  terms  of 
instrument  readings.  The  full  spectrum  visual/motion  simulation  capability  of 
the  ASPT  allows  extension  Into  contact  flying.  Including  aerobatics  and  forma- 
tion. Innovative  recording  formats  can  be  safely  and  efficiently  developed  and 
refined  before  validation  in  flight,  and  potential  measurement  techniques  can 
be  tested  against  the  criteria  of  IP  observability,  recordability , workload, 
and  safety.  This  could  be  a critical  st.n  as  IPs  might  accept  as  useful  a 
technique  learned  first  In  the  trainer  and  then  used  in  the  airplane,  whereas 
they  might  reject  It  if  they  encountered  iv  for  the  first  time  under  the 
stresses  of  a training  flight. 

Such  a program  can  provide  a "family"  of  measuring  instruments  to  support 
training  methods,  research,  and  hardware  evaluations  which  require  more  pre- 
cise data  than  existing  subjective  measures  allow. 
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Tha  function  of  Chl*  book! at  la  ca  prcvida  4 raoord  for  acudamt 
fmrtormmam  la  flying  bit  Tartieal  t-A  Mattntr.  Tha  booklat  coualat* 
of  t aarlaa  of  acalaa  on  which  tha  pa  .fornaaea  It  r»  cor  dad . Shown  balow 
la  aa  tu^lt  of  4 typical  icalai 


For  our  purpoini  cha  Vartical  S-A  has  baan  brokio  loco  fiva  lagaanta  : 
Transition  to  Cllrt,  Clinb,  Tranaltioo  to  Daacaat,  Doacanc  and  Laval-Off, 

All  tha  icalaa  (axcapt  for  Haadlag)  in  coaacructnd  to  abaoluct  valuaa,  a.g. 
cha  Alrtpaad  a cal  a goal  fro*  140  to  180  In  2 CAS  intarvala.  Tha  haadlag 
aeala  goal  fron  4 taro  poiae  (In  tha  cantar  of  tha  icala)  and  thowa 
darlacLona  la  a poaltivu  dlraccioa  (+3  thro  ■*■20)  and  la  a nagatlva  41  net  Ion 
(-3  thro  -20).  Tha  haadlag  aaiuaad  by  tha  atulaar  will  ba  coeildsrad  tha 
*aro  point  and  variation!  will  ba  raeordad  fro*  that  haadlag. 
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Table  D-l 

Analysis  of  Variance:  Heading  Deviation  Range 


Source  of 
Variance 

Sum  cf 
Squares 

df 

Mean 

Square 

F 

£ 

Groups  (G) 

5.14 

2 

2.57 

<1 

ns 

Ss:  G 

152.75 

27 

5.66 

- 

- 

Trial  (T) 

39.96 

5 

7.99 

3.34 

< .01 

T X G 

10.59 

10 

1.06 

< 1 

ns 

Ss  X T 

322.95 

135 

2.39 

< 1 

ns 

Table 

D-2 

Analysis  of 

Variance: 

Airspeed  Deviation  Range 

Source  of 
Variance 

Sum  or 
Squares 

d If 

Mean 

Square 

F 

£ 

Groups  (G) 

23.47 

2 

11.74 

< 1 

ns 

Ss:  G 

362.55 

27 

13.43 

- 

Trials  (T) 

116.16 

5 

23.63 

11.53 

< .01 

T X G 

23.26 

10 

2.33 

1.14 

ns 

Ss  X T 

276.75 

135 

2.05 

- 

- 

Table  D-3 
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Analysis  of  Variance:  Vertical  Velocity  Deviation  Range 


Source  of 
Variance 

Sum  of 
Squares 

11 

Mean 

Square 

F 

£ 

Groups  (G) 

120.46 

2 

60.23 

11.23 

< .01 

Ss:  G 

144.71 

27 

5.36 

- 

- 

Trials  (T) 

9.54 

5 

1.91 

3.60 

< .01 

T X G 

18.17 

10 

1 .82 

3.43 

< .01 

Ss  X T 

70.89 

135 

0.53 

- 

- 

Table  D-4 

Analysis  of  Variance:  Time  Maximum  Deviation 

Source  of 

Sum  of 

Mean 

Variance 

Squares 

df 

Square 

F 

£ 

Groups  (G) 

5324.13 

2 

2662.06 

7.73 

< .01 

Ss;  G 

9139.15 

27 

338.49 

- 

- 

Trials  (T) 

573.32 

5 

114.66 

2.34 

A 

O 

cn 

T X G 

2661.80 

10 

266.18 

5.44 

<.01 

Ss  X T 

6611.55 

135 

49.97 

_ 

Table  D-5 

Analysis  of  Variance:  Maximum  Altitude  Deviation 
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Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

F 

£ 

Groups  (G) 

8.72 

2 

4.36 

<1 

ns 

Ss  X G 

110.53 

27 

4.39 

- 

- 

Trials  (T) 

8.99 

5 

1.80 

<1 

ns 

T X G 

28.61 

10 

2.86 

1.24 

ns 

Ss  X T 

312.07 

135 

2.31 

- 

- 

Table  D-6 

Analysis  of  Variance:  First  Lead  Point  Maximum  Deviation 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Groups  (G) 

188.81 

2 

94.41 

3.87 

Ss;  G 

658.75 

27 

24.40 

- 

Trials  (T) 

54.89 

5 

10.98 

1 .89 

T X G 

90.13 

10 

9.01 

1.55 

Ss  X T 

783.15 

135 

5.80 

£ 


Table  D-7 


Source  of 
Variance 

Sum  of 
Squares 

df 

Groups  (G) 

79.64 

2 

Ss:  G 

362.52 

27 

Trials  (T) 

16.56 

5 

T X G 

44.76 

10 

Ss  X T 

518.18 

135 

Mean 

Square  F £ 

39.32  2.97  < ,1Q 

13.43 
3.31 
4.48 
3.84 


